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Why this study? 


There is widespread agreement that improving high school graduation rates is a high prior- 
ity for education reform. While reported graduation rates have risen nationwide in recent 
years, nearly one of every five students who begins 9th grade in this country fails to earn 
a high school diploma within the next four years (National Center for Education Statistics 
[NCES], 2016). Graduation rates vary substantially across districts and states, and, compared 
to the rates for White students, graduation rates are substantially lower among Black and 
Hispanic students. Similarly, students from disadvantaged families have lower graduation 
rates than their more economically well-off peers. Nationally, the graduation rates for Black 
and Hispanic students are 73 and 76 percent, respectively, compared to 87 percent for White 
students (NCES, 2016). In Nevada, graduation rates are 71 percent overall, 56 percent for 
Black students, 67 percent for Hispanic students, 64 percent for economically disadvantaged 
students, and 32 percent for students with limited English proficiency (NCES, 2016). 

Clark County School District (CCSD), in Las Vegas, Nevada, is the largest school district 
in the state and the fifth largest in the country. Like many other large urban school districts, 
CCSD has identified improving graduation rates as one of its core goals. This report 
emerges from collaborative efforts between CCSD and the Regional Educational Laboratory 
West (REL West) around strategies for using predictive analytics to help achieve that goal. 

In particular, CCSD is interested in finding early indicators that can help identify students 
at risk of not graduating. The district also wants to better understand how students’ atti- 
tudes, beliefs, and behaviors predict school success in the transition to high school and, thus, 
how they might relate to graduation rates. 

CCSD is a particularly useful place to study factors related to graduation rates. Large urban 
school districts like CCSD educate a disproportionately large number of English learner 
students, economically disadvantaged students, Black students, and Hispanic students (Sable, 
Plotts, & Mitchell, 2010). A better understanding of the dynamics around student progress 
through school, the early identification of students who are at risk of failing to graduate on 
time or not graduating at all, and an understanding of the associations between students’ 
academic mindsets and behaviors and students’ transition to high school may better position 
policymakers and practitioners to develop, refine, and test interventions aimed at improving 
9th grade success in these contexts. 

This study explored two issues related to graduation: early warning indicators and the relation 
of academic mindsets to students’ progress through school. The first part of the report focuses 
on early indicators of student risk of not graduating from high school, while the second part 
of the report addresses the relationship between student self-reports of academic mindsets 
and behaviors in the 8th grade on students’ on-track status in the 9th grade. 

Early warning indicators 

As part of their efforts to improve graduation rates, many districts have developed and 
implemented on-track, or early warning, systems to identify students who are at risk of 
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failing to graduate. These efforts have built on previous research that has identified early 
outcomes that are effective indicators of whether or not students are likely to graduate from 
high school on time. The available evidence shows that risk indicators such as low atten- 
dance, failure to earn enough credits to progress from one grade to the next, receiving Fs, 
and disciplinary actions, such as suspensions, are significant predictors of students’ failure to 
graduate on time (Allensworth & Easton, 2005, 2007; Balfanz, Herzog, & Maclver, 2007; 
Balfanz & Legters, 2004). This evidence has been used to develop a variety of early warning 
approaches, including indicators that identify whether or not students are on track for grad- 
uation as they reach the end of the 9th grade. 

Like many districts, CCSD has developed a set of indicators of students’ on-track status 
during the 9th grade. In fact, the district has developed benchmarks for adequate progress 
in every year of high school. But because the available research suggests that students who 
are off track at the end of the 9th grade are highly unlikely to recover and graduate within 
the standard four-year period (Allensworth & Easton, 2005, 2007), CCSD has also sought 
to explore the extent to which it could identify whether students were at risk of falling off 
the graduation track before the 9th grade. To the extent this is possible, the district could 
share those early risk indicators with its schools, so that the schools, in turn, could identify 
and support students who are at risk of falling off track as they make the transition to high 
school, rather than waiting until the students have already completed the 9th grade and 
perhaps fallen even further behind. 

To help CCSD explore the possibility of identifying students at risk prior to students enter- 
ing the 9th grade, REL West used CCSD’s student records data to answer the following 
research questions: 

1. To what extent can 8th grade indicators of attendance, credits earned, Fs, and 
student grade point average (GPA) identify 8th grade students at risk of falling off 
track for graduation by the end of the 9th grade? 

2. How do 9th grade on-track rates among students who were identified in the 
8th grade as being at risk of falling off track vary depending on which combina- 
tions of 8th grade indicators were used to identify students at risk? 

3. How does the percentage of 9th grade off-track students who were correctly iden- 
tified in the 8th grade as being at risk of falling off track, as well as the percentage 
of 9th grade students who had been incorrectly identified as likely to be off track, 
vary depending on which combinations of 8th grade indicators are used to classify 
students as being at risk of falling off track? 

Academic mindsets and behaviors 

In addition to identifying 8th grade students who are likely to fall off track for graduation 
by the end of the 9th grade, CCSD is interested in exploring the underlying factors poten- 
tially driving students’ school success or failure in high school. A growing body of research 
suggests that students’ beliefs, attitudes, and dispositions (that is, their academic mind- 
sets) affect their engagement and participation in school (that is, their academic behaviors ), 
thereby having a substantial effect on their academic outcomes (Dweck, Walton, & Cohen, 
2011; Farrington et ah, 2012; Snipes, Fancsali, & Stoker, 2012). As part of their effort to 
understand these dynamics and how they play out in CCSD schools, beginning in the 
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2014/15 school year, CCSD staff started collecting survey data regarding several dimensions 
of academic mindsets and behaviors hypothesized to contribute to variation in academic 
engagement and progress. On annual surveys administered to students in grades 4-12 
throughout the district, CCSD included measures of students’ growth mindset (that is, belief 
in the malleability of ability and performance and in the payoff to academic effort), perfor- 
mance avoidance (that is, hiding one’s effort or refraining from making an effort due to 
concerns about failing or being embarrassed), and academic behaviors (such as completing 
homework and participating in class) (see box 1 on page 4). 

Previous analysis of these self-reported data show that students’ scores on survey measures 
of academic mindsets and behaviors varied significantly across students based on ethnicity, 
English learner status, and prior achievement (Snipes & Tran, forthcoming). With this in 
mind, the district was interested in examining the extent to which these survey measures 
predicted students’ future academic success, including progress through high school. Such 
an analysis might shed light on potential targets for intervention among students at risk 
of failing to graduate, as well as validate the academic relevance of the district’s survey 
measures. To support this inquiry, REL West used the CCSD data to attempt to answer the 
following research question: 

4. To what extent do 8th grade measures of growth mindset, performance avoidance, 
and academic behavior predict differences in the probability of students being on 
track at the end of the 9th grade? 

This report presents the results of the analyses related both to early warning indicators 
and to academic mindsets. The following section reviews the methods and findings for the 
analysis of early indicators and the prediction of 9th grade on-track status, while the third 
section presents the methods and findings regarding the analysis of the relationship between 
academic mindsets and behaviors in the 8th grade and on-track status in the 9th grade. The 
final section presents conclusions and potential implications from the analysis. 
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Box 1. Key terms used in this report 


Academic mindsets. Student attitudes, beliefs, perceptions, and disposition regarding 
themselves, their academic potential, and their relationship to school (Dweck, Walton, & 
Cohen, 2011). 

Academic behaviors. Behaviors commonly associated with being a “good student,” 
including arriving ready to work (with necessary supplies and materials), regularly 
attending class, paying attention and participating in class, and devoting out-of- school 
time to studying and completing homework (Farrington et ah, 2012). 

Fixed-ability mindset. A dimension of academic mindsets, also referred to as an 
“entity” theory of intelligence. The belief that intelligence and academic ability are 
fixed qualities that students either do or do not possess and that they cannot be changed 
through effort (Dweck, Walton, & Cohen, 2011). 

Growth mindset. A dimension of academic mindsets. The belief that intelligence and 
academic ability are not fixed but are malleable and can be increased through effort and 
learning (Dweck, Walton, & Cohen, 2011). 

Off-track status. An indicator of whether students have fallen off course for on-time 
graduation at the end of the 9th grade. For this analysis, students were considered to 
be off track at the end of the 9th grade if they either failed to receive enough credits to 
progress to the 10th grade (fewer than 6), or if they received more than one semester F. 
Otherwise they were considered on tracks 

Performance avoidance. A dimension of academic mindsets. The tendency to change 
behavior — for example, hiding effort and avoiding academically challenging situations — 
because of concerns about failure or embarrassment. The concept is related to, but 
distinct from, holding a fixed-ability mindset (Farrington, Levenstein, & Keyes, 2014). 

a. CCSD has also developed its own definition of on-track status. According to this indicator, a 
student is only on track if, at the end of the year, he or she successfully earns a full-year credit in 
each of the required core subjects and an elective, earns half-credits for courses on computer use 
and health, and successfully passes end-of-course exams for English Language Arts I, English 
Language Arts II, Mathematics I, and Mathematics II. Appendix A includes an analysis of the 
relationship between indicators and this on-track variable. 
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Early warning indicators for 
off-track status 



Analytic approach 

The goal of this first analysis was to identify early indicators of whether 8th grade students 
were likely to fall off track for graduation by the end of the 9th grade and to examine the 
extent to which different 8th grade indicators accurately predicted 9th grade off-track status. 

Sample 

To conduct this analysis, REL West employed administrative records collected by CCSD in 
the 2014/15 and 2015/16 school years. The initial CCSD sample consisted of 26,322 students 
who were enrolled in the 8th grade in CCSD during the 2014/15 school year and who 
were also enrolled in CCSD in the 9th grade during the 2015/16 school year. Students for 
whom the required data for the analysis were not available were eliminated from the sample, 
resulting in a final analytic sample of 22,335. The characteristics of the students in this 
sample were largely similar to those of students in the district as a whole (see appendix A for 
more details). 

Defining off-track status 

Previous studies have identified specific criteria for whether or not students are off track 
for graduation at the end of the 9th grade. Allensworth and Easton (2005, 2007) classified 
students as being off track for graduation at the end of the 9th grade if they 

» had failed to earn sufficient credits for promotion from the 9th to the 10th grade; or 
» had received more than 1 semester F in a core subject that year. 

These studies found that students who were off track at the end of the 9th grade were 
substantially less likely than their on-track counterparts to graduate from high school in 
four years. Moreover, on-track status predicted graduation more effectively than did prior 
achievement tests or student demographic characteristics. Although there is some variation 
in these patterns across different school districts (Hartman et ah, 2011), the basic patterns 
appear to be consistent across a variety of contexts (Stuit et ah, 2016). Following this defini- 
tion, students in the sample for these analyses were classified as being off track at the end of 
the 9th grade if they either had earned insufficient credits for promotion (that is, fewer than 
6 credits) 1 or had received more than one semester F during the course of their 9th grade 
year (see box 1 on page 4). 

Creating 8th grade risk indicators 

A primary goal of these analyses was to ascertain the extent to which the available data 
might allow CCSD to identify 8th grade students who, without any intervention, are likely 


1 It is not clear whether students who fail to complete the number of credits they are expected to 
earn in the 9th grade are actually kept from progressing to the 10th grade in CCSD. 
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to struggle and fall off track in the 9th grade. The research suggests that attendance, credits 
earned, grade point average (GPA), and the number of Fs students accumulate in earlier 
grades are significant predictors of progress thorough high school and on-time graduation 
(Allensworth & Easton, 2005, 2007; Balfanz & Legters, 2004; Balfanz, Herzog, & Mac Iver, 
2007). Based on these findings, REL West used CCSD student records data to create dichot- 
omous 8th grade risk indicators applied to each 8th grade student in the sample signaling 
whether or not that student had: low attendance; insufficient credits; low GPA; or one or 
more Fs (see box 2). These 8th grade risk indictors were then examined individually and 
in combination with one another to ascertain the extent to which they identified 8th grade 
students (in 2014/15) who fell offtrack during the following school year (2015/16) and could 
therefore be used as early warning indicators. 


Box 2. Eighth-grade risk indicators 


Low-attendance rate. Previous research has shown that having an attendance rate 
below 90 percent is a significant predictor of academic progress and high school gradu- 
ation (Balfanz, Herzog, & Mac Iver, 2007). Having an attendance rate below 90 percent 
during the 8 th grade was considered an indicator of risk. 

Insufficient credits. Previous research identifies whether students earn sufficient credits 
to make progress from one grade to the next as a key predictor of high school success. 
For this study, CCSD 8th grade students were classified according to whether or not 
they accumulated enough credits to make adequate progress from one grade to the next 
(six credits). Accumulating fewer than six credits over the course of the 8 th grade was 
considered an indicator of risk. 

Low grade point average. Student grade point averages were calculated across all class- 
es for which students received letter grades. a Having a grade point average below 2.0 in 
the 8 th grade was considered an indicator of risk. 

Semester Fs. Students were categorized according to the total number of F grades they 
received in each semester of the 8th grade. Receiving one or more semester Fs was con- 
sidered an indicator of risk. 

a. Point values were assigned as follows: A = 4.0; B = 3.0; C = 2.0; D = 1.0; F = 0. Pass/fail classes 
and classes with marks other than letter grades were not included in the calculation of student GPA. 


In addition to analyzing the accuracy of using individual risk indicators to identify students 
at risk, REL West examined the extent to which using composite variables that combined 
multiple risk indicators aided in the accurate identification of students at risk of falling 
off track. Among the composite indicators REL West created were several that identified 
students as being at risk based on the total number of risk indicators they demonstrated 
in the 8th grade. These included flags for whether students had at least one 8th grade risk 
indicator for attendance, credits, Fs, or low GPA, as well as flags for whether students 
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demonstrated two or more or three or more of these risk factors. REL West also created 
different composite indicators that flagged students if they had: at least one 8th grade risk 
indicator for attendance, credits, or Fs; at least two 8th grade risk indicators for attendance, 
credits, or Fs; at least one 8th grade indicator for credits, Fs, or low GPA; and at least two 
8th grade indicators for credits, Fs, or low GPA. 

Defining accuracy of the risk indicators 

Determining the accuracy of these potential early warning indicators was an important goal 
for this analysis. For that purpose, two different rates were examined for each one: 

1. its “true-positive” rate; and 

2. its “false-positive” rate (see box 3). 

The true-positive rate 2 answers the following question: Among those students who actually 
fell off track by the end of the 9th grade, what percentage would have been flagged as at 
risk based on the given 8th grade indicator? As represented graphically by the space in the 
upper right quadrant in figure 1, if a student demonstrated a particular 8th grade risk indi- 
cator and by the end of the 9th grade that same student had fallen off track, then using this 
indicator to predict off-track status would result in a correct prediction for that student — a 
true-positive. If on the other hand, a student did not have that particular 8th grade risk 
indicator, but nonetheless fell off track by the end of the 9th grade, then using that indicator 
would result in a false-negative for that student, as represented by the lower right quadrant 
in figure 1. So, if 25 of 100 students who fell off track by the end of the 9th grade had a 
particular 8th grade risk indicator, then using that indicator to classify students as at risk of 
falling off track would result in a true-positive rate of 25 percent (and a false-negative rate 
of 75 percent). 

The false-positive rate (the upper left quadrant in figure l), on the other hand, answers the 
following question: Among students who did not fall off track by the end of the 9th grade, 
what percentage would have been categorized as being at risk for doing so? 3 For example, if, 
out of a total of 100 students who did not fall off track by the end of the 9th grade, 25 had 
a particular 8th grade risk indicator, the false-positive rate resulting from using that indi- 
cator would be 25 percent (the true-negative rate-represented by the lower left quadrant of 
figure 1-would be 75 percent). 


2 This is sometimes referred to as the “correct-prediction” rate (Stuit et ah, 2016). 

3 Other analyses have pursued “area under the curve” approaches to defining accuracy. Such 
an analysis was not included in this report. Area-under-the-curve analyses make implicit value 
judgments about the relative costs and benefits of correctly identifying students who will fall off 
track versus categorizing students who did not actually fall off track as being at risk. Since those 
tradeoffs are essentially policy and practice decisions that might be made differently depending on 
how the stakeholders involved view these costs and benefits of each of these outcomes, REL West 
did not include this type of analysis in the report. 
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Figure 1. Defining accuracy of the risk indicators: Correct prediction 
versus false alarm rates 
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Box 3. On-track, true-positive, and false -positive rates 


On-track rate. Percentage of students who were on track for graduation at the end of 
the 9th grade. 

True-positive rate. Proportion of students who are not on track by the end of the 
9th grade and who had been identified correctly in the 8th grade as being at risk of fall- 
ing offtrack by that time. For example, if 100 8th grade students were flagged as being 
at risk of falling off track by the end of the 9th grade because they had a grade point 
average (GPA) below 2.0, and 25 of those 100 students did, in fact, fall offtrack by that 
time, then that particular risk indicator (that is, having a GPA less than 2.0) would have a 
true-positive rate of 25 percent (Stuit et ah, 2016). 

False -positive rate. Percentage of students who are on track in grade 9 but who had 
been incorrectly flagged by an 8 th grade indicator as being at risk of being off track. 

If 100 students remained on track at the end of the 9th grade, and 25 of them had an 
8th grade GPA below 2.0, the false alarm rate for the GPA indicator would be 25 percent. 
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Findings 

Seventy-one percent of 8th grade CCSD students were on track at the end of the 9th grade 

REL West calculated the percentage of 8th graders from the 2014/15 school year who were 
on track at the end of the 9th grade the following school year (2015/16). As illustrated in 
figure 2, 71 percent of CCSD students who had been 8th graders in school year 2014/15 
were on track for high school graduation at the end of 9th grade, while 29 percent of these 
students were off track for graduation by this point in time. 


Figure 2. Percent of CCSD students on track and off track at the end 
of the 9th grade (2015/16) 



Source: Authors’ analysis of primary data collected for the study. Sample size = 22,335. 


Ninth-grade on-track rates declined as the number of 8th grade risk indicators went up. 

The more 8th grade risk indicators students had, the lower their chances of being on track 
at the end of the 9th grade. Among 9th grade students with no 8th grade risk indicators, 
on-track rates at the end of the 9th grade were 88 percent (figure 3). Among students with 
only one 8th grade risk indicator, on-track rates at the end of the 9th grade were 74 percent. 
However, among 9th grade students with two, three, or four 8th grade risk indicators, 
on-track rates at the end of the 9th grade were 60, 34, and 19 percent, respectively. 
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Figure 3. Ninth-grade (2015/16) on-track rates by number of 
8th grade (2014/15) risk indicators 


100 
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N: 0 = 12,636; 1 = 2,793; 2 = 2,179; 3 = 3,137; 4 = 1,590 

Source: Authors’ analysis of primary data collected for the study. Sample size = 22,335. 


The majority of students who fell off track by the end of 9th grade had displayed multiple risk 
indicators in the 8th grade. 

The analysis of risk indicators among students who fell off track shows that the vast major- 
ity (77 percent) had demonstrated at least one of the four 8th grade risk indicators identified 
above (see figure 4), while 23 percent had shown none of these indications of risk at the 
end of the 8th grade. Eleven percent had a single risk indicator. However, approximately 
two thirds (66 percent) of the students who fell off track by the end of 9th grade had two or 
more 8th grade risk indicators. In fact, more than half of those who fell off track (52 percent) 
demonstrated three or more risk indicators during the 8th grade. This suggests that the 
majority of CCSD students who fell off track in the 9th grade were already experiencing a 
variety of academic challenges before they even arrived at high school. 
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Figure 4. Number of 8th grade risk indicators demonstrated by 
students who fell offtrack in 9th grade (2015/16) 



Source: Authors’ analysis of primary data collected for the study. Sample size = 6,470. 


Indicators for low attendance, insufficient credits, Fs, and low GPA are all associated with lower 
on-track rates. 

REL West calculated the 9th grade on-track rates associated with four 8th grade risk indica- 
tors: low attendance (below 90 percent), insufficient credits (fewer than six), low GPA (lower 
than 2.0), and receiving one or more Fs during the 8th grade (see figure 5). Each of these 
8 th grade indicators was associated with an elevated risk of being offtrack. While the aver- 
age 9th grade on-track rate for this cohort of 8th graders was 71 percent, average on-track 
rates among students with any of the 8th grade risk indicators ranged between 30 and 
45 percent. The indicator for low GPA was associated with an on-track rate of 30 percent, 
and the indicator for receiving one or more Fs was associated with an on-track rate of 
39 percent. Grade 8 students who earned fewer than six credits had a 9th grade on-track rate 
of 43 percent, and those with low attendance had an on-track rate of 45 percent. 
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Figure 5. Eighth-grade risk indicators and 9th grade on-track rates 
(2015) 
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Source: Authors’ analysis of primary data collected for the study. Sample size = 22,335. 


Compared to attendance, academic performance indicators correctly identified a larger percentage 
of the students who actually fell off track the following year. 

Compared to the indicator for low attendance, risk factors related to academic perfor- 
mance captured a higher percentage of the students who fell off track at the end of the 
9th grade (table l). As explained above, the true-positive rate captures the percentage of 
9th grade off-track students who were correctly identified by a given 8th grade indicator. 
The true-positive rate for the credit indicator was approximately twice as high as the true- 
positive rate for the attendance indicator (64 versus 32 percent). In other words, compared 
to the attendance indicator, the credit indicator accurately identified twice as many of the 
8th grade students who fell offtrack by the end of the 9th grade. The relatively low percent- 
age of off-track students identified by attendance alone suggests that the effectiveness of 
using attendance as an indicator may be due in part to the fact that it is associated with other 
academic outcomes, and that there are substantial numbers of students who fall off track at 
the end of the 9th grade who did not have low attendance in the 8th grade. 

The indicators for low GPA and for whether 8th grade students earn at least one F, when 
used individually, also identified the majority of students who fell off track in the 9th grade. 
The 9th grade on-track rate among students who earned at least one F during the 8th grade 
was 39 percent, and using this characteristic alone to identify at-risk students captured 
65 percent of the 8th graders who fell on track the next year. The 9th grade on-track rate 
among students who have a GPA of less than 2.0 in the 8th grade was only 30 percent, and 
this characteristic alone identified 55 percent of students who fell off track by the end of the 
9th grade. The false-positive rates, that is, the percentage of students who ended up being 
on track at the end of the 9th grade despite being classified as at risk based on insufficient 
credits, low GPA, or receiving Fs, ranged from 10 to 20 percent. 
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Table 1. True-positive and false -positive rates for 8th grade indicators 


8th grade indicator 

True -positive rate 

False-positive rate 

Attendance <90% 

31.6 

10.5 

Credits <6 

63.9 

19.8 

One or more Fs 

64.5 

17.1 

GPA < 2.0 

54.6 

9.6 

Composite indicators 

1. Attendance, Credits, or Fs 

75.0 

28.9 

2. Attendance, Credits, or Fs (>=2 indicators) 

62.0 

15.5 

3. Attendance, Credits, Fs, or Low GPA 

76.9 

29.8 

4. Attendance, Credits, Fs, or Low GPA (>=2 indicators) 

65.6 

16.8 

5. Attendance, Credits, Fs, or Low GPA (>=3 indicators) 

52.1 

8.6 

6. Credits, Fs, or Low GPA 

72.7 

24.7 

7. Credits, Fs, or Low GPA (>=2 indicators) 

62.7 

14.7 


Note: The true-positive rate for each indicator is the number of students who had demonstrated that indica- 
tor in the spring of the 8th grade (2014/15) and who were offtrack in the spring of the 9th grade (2015/16), 
divided by the total number of students who were offtrack in the spring of the 9th grade (2015/16). The 
false-positive rate for each indicator is the number of students who had demonstrated that indicator in the 
spring of the 8 th grade (2014/15), and who were on track in the spring of the 9th grade (2015/16), divided by 
the total number of students who are on track in the spring of the 9th grade (2015/16). 

Source: Authors’ analysis of primary data collected for the study. Sample size = 22,355. 


Used in combination with one another, 8th grade indicators identified more than three quarters of 
the students who eventually fell off track in the 9th grade. 

The analysis examined on-track rates, true-positive rates, and false-positive rates for differ- 
ent combinations of 8th grade indictors. These “composite indicators” were generally more 
effective than the individual indicators at accurately identifying 8th grade students at risk 
of falling off track at the end of the 9th grade. For example, students who had at least one of 
the following characteristics— low attendance, insufficient credits, or one or more Fs— had 
an on-track rate of 49 percent (see figure 5). This was somewhat higher than the on-track 
rates generated by using individual 8th grade risk indicators. However, the true-positive 
rate for this composite indicator shows that using this approach identified 75 percent of the 
students who fell off track in the 9th grade. If an indicator for whether or not a student had 
an 8th grade GPA below 2.0 was added, the percentage of the students who eventually fell 
offtrack that were identified at the end of the 8th grade increased to 77 percent (table l). In 
short, a simple indicator based on whether or not a student met any of four criteria at the 
end of the 8th grade appears to identify more than three quarters of the students who will 
fall off track in the 9th grade, before they enter high school. 
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There is a tradeoff between increased identification of off-track students and increased numbers of 
false positives. 

With true-positive rates of 75 and 77 percent, respectively, the composite indicator for 
attendance, credits, or Fs, and the composite indicator for attendance, credits, Fs, or low 
GPA correctly identify the largest share of 8th graders who eventually fall off track in the 
9th grade. The false-positive rates, on the other hand, capture the percentage of students 
who did not fall off track, but who, based on the 8th grade indicators, had been identified as 
being at risk of falling off track. The false-positive rates for these composite variables were 
29 and 30 percent, respectively (table l). This suggests that if students were to be catego- 
rized as at risk based on these indicators, just under a third of those who did not actually fall 
off track would have been identified as being at risk for doing so. The false-positive rates 
were lower for indicators requiring that students demonstrate more than one risk factor 
before being classified as at risk. For example, using the composite variable identifying 
students as being at risk of falling off track only if they have demonstrated at least one of the 
four potential individual risk indicators (that is, low attendance, insufficient credits, receiv- 
ing Fs, and low GPA), resulted in a false-positive rate of 29 percent. Using the composite 
variable that categorized student as at risk if they had at least two of these same individual 
risk indicators reduced the false-alarm rate from 29 to 17 percent (table l). However, this 
approach also caused the percentage of the off-track students who had been successfully 
identified to fall from 75 to 66 percent. In general, if students needed to demonstrate more 
risk indicators before being classified as at risk, the rate at which they were “incorrectly” 
identified as being at risk fell, but so, too, did the proportion of truly at-risk students who 
were correctly identified. Policymakers and practitioners executing early warning systems 
would have to weigh these tradeoffs when deciding on their preferred early indicators. 

Composite indicators identify larger numbers of students and generate larger numbers of 
true-positives and false-positives. 

One way to think about the potential trade-offs between different indicators is to examine 
the implications for the average number of students per school that were identified, as well 
as the number of true-positives and false-positives, using each potential indicator (table 2). 
There are 49 high schools in the Clark County School District. The CCSD administrative 
records data used for this analysis suggest that there were approximately 22,335 students 
in the 2015/16 class of CCSD 9th graders for whom CCSD would have indicator data. 4 
Though the sizes of the entering classes likely vary across schools and as a result of students 
entering from outside of the district, in terms of students coming from CCSD middle 
schools (those for whom these indicators are likely to be available) these calculations suggest 
an average entering class of approximately 455 9th graders per CCSD high school. 

Based on the percentages of students in the sample who were identified by using each of 
the indicators in table 1, REL West calculated the average number of incoming 9th graders 
at each school that would be identified as at risk using each of the different risk indicators 
discussed in this report (table 2). The estimated number of incoming students identified based 
on these indicators varied from a low of 32 using the attendance indicator, to a high of 211 

4 This does not include students who were enrolled in the 9th grade, but left the school before the end 
of the year, or students who entered CCSD from another district. The analysis assumes that the district 
would not have readily available transcript data for incoming transfers in time to conduct this analysis. 
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using the composite indicator based on attendance, credits, Fs, and GPA (table 2). REL West 
also calculated the average number of true-positives and false-positives that would be 
observed based on the current sample. The results suggest that the indicators that identify the 
largest numbers of true-positives are the same indicators that identify the largest numbers of 
false-positives. Specifically, the composite indicator based on attendance, credits, Fs, and GPA 
is expected to generate an average of 152 true-positives, that is, students who without any 
additional intervention would indeed fall off track. At the same time, this indicator would be 
expected to generate an average of 59 false-positives, that is, students who appear to be at risk 
but who, without any additional intervention, would not actually fall off track. 

Schools may be able to use this information to plan and deploy counseling resources, as 
well as to weigh the tradeoffs between different indicators. As they do so, it would be useful 
to keep in mind that though many of the false-positive students might not actually fall off 
track at the end of the 9th grade, based on their 8th grade outcomes, they still may have an 
elevated risk of academic challenges throughout high school. As such, supports and inter- 
ventions targeted to these students may still be useful expenditures of resources. 


Table 2. Average numbers of identified students, true-positives, and 
false-positives for CCSD high schools 


8th grade indicator 

Total incoming 
students identified 
(per school) 

True- 
positives 
(per school) 

False- 
positives 
(per school) 

Overall on-track rate 

Attendance <90% 

32 

24 

8 

Credits <6 

124 

95 

29 

One or more Fs 

115 

91 

24 

GPA < 2.0 

66 

56 

10 

Composite indicators 

1. Attendance, Credits, or Fs 

200 

144 

56 

2. Attendance, Credits, or Fs (>=2 indicators) 

102 

82 

20 

3. Attendance, Credits, Fs, or Low GPA 

211 

152 

59 

4. Attendance, Credits, Fs, or Low GPA (>=2 indicators) 

116 

92 

24 

5. Attendance, Credits, Fs, or Low GPA (>=3 indicators) 

58 

50 

8 

6. Credits, Fs, or Low GPA 

172 

128 

44 

7. Credits, Fs, or Low GPA (>=2 indicators) 

101 

82 

19 


Source: Authors’ analysis of primary data collected for the study. Sample size = 22,355. 

Notes: 

Students identified per school is calculated by taking the total sample of 22,335, dividing that number by 
the total number of CCSD high schools (49), and multiplying the result (455) by the average percentage of 
students categorized as at risk of falling off track using each risk indicator. 

True-positive students per school is the average number of incoming 9th grade students per school who had 
the 8th grade indicator in 2014/15 but who were offtrack in the spring of the 9th grade (2015/16). 

False-positive students per school is the average number of incoming 9th grade students per school who had 
the 8th grade indicator in 2014/15 but who were not offtrack in the spring of the 9th grade (2015/16). 
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Academic mindsets and behaviors and 
9th grade on-track status 



Analytic approach 

In addition to identifying 8th grade risk factors associated with being off track in the 
9th grade, CCSD staff were interested in understanding some of the underlying student 
attitudes and behaviors that might be associated with 9th grade success. While these factors 
might not be appropriate to use as indicators of risk (Duckworth, 2016), an analysis of their 
relationship to 9th grade success might shed some light on potential strategies for student 
support. 5 Therefore, the goal of this analysis was to examine the relationship between 
8th grade student self-reports of academic mindsets and behaviors and their on-track status 
at the end of the 9th grade. 

Sample 

In the spring of the 2014/15 school year, CCSD administered surveys to students in 
grades 4-12 throughout the district. These surveys included a series of questions regarding 
academic mindsets and behaviors. 6 The sample for the analysis of the relationship between 
the survey measures of students’ academic mindsets and behavior and their on-track status at 
the end of 9th grade consists of the 13,578 students who were included in the initial analytic 
sample of early indicators and for whom 8th grade survey measures of academic mindsets 
and behaviors were obtained in the spring of 2014/15. Although there were some minor 
differences, the characteristics of the students in this sample were largely similar to those of 
the students in the district as a whole (see appendix A). 

Defining academic mindsets and behaviors 

The survey data included three measures of academic mindsets and behaviors: growth mind- 
set (that is, believing in the malleability of ability and in the payoffs of academic effort); 
performance avoidance (that is, hiding one’s effort or refraining from making an effort due 
to concerns about failure or embarrassment); and academic behaviors (such as completing 
homework and participating in class) (see table 3). Previous research suggests that these 
academic mindsets and behaviors are strongly linked to student success, and that interven- 
tions targeting these beliefs can have significant effects on student outcomes (Farrington 
et ah, 2012; Snipes, Fancsali, & Stoker, 2012; Yeager, Walton, & Cohen, 2013). The analysis 
presented below used logistic regression to examine the association between these measures 
of academic mindsets and behaviors reported during the spring of students’ 8th grade school 


5 To the extent that survey responses to question regarding student attitudes, beliefs, and 
behaviors results in the identification of students for additional attention and support, and students 
become aware of this, future responses to these survey items can become distorted to reflect 
students’ desires to either receive or avoid this attention. 

6 These measures were adapted from the Becoming Effective Learners Student (BEL-S) survey, 
originally developed by the Chicago Consortium for School Research (Farrington & Levenstein, 
2013). 
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year and the on-track status of those same students at the end of the 9th grade. (For more 
information on the study’s use of logistic regression, see appendix C.) 


Table 3. Clark County School District survey measures and internal 
consistency statistics, 2015 


Measure 

Student version 


Questions 

Reliability 3 

Growth mindset 

How true are the following about you? 

1. My intelligence is something that I can’t change very much. 

2. Challenging myself won’t make me any smarter. 

3. There are some things I am not capable of learning. 

4. If I am not naturally smart in a subject, I will never do well in it. 

Not at all true, A little true, Somewhat true, Mostly true, Completely true 
[reverse coded 5-1 ] 

.75 

Performance 

avoidance 

In a typical class, how true are the following? 

1. I don’t participate in discussions because I am afraid people might think I 
am foolish. 

2. I would rather do easy work that I can do well than challenging work 
where I might learn more. 

3. I don’t ask questions in class because people might think my questions are 
not smart. 

4. I stop doing work if I feel like I can’t do it well. 

5. I only volunteer to answer a question if I am sure my answer is right. 

Not at all true, A little true, Somewhat true, Mostly true, Completely true [coded 1-5] 

.77 

Academic behaviors 

In a typical class, how often do you: 

1. Do the readings or other assigned work to prepare for class. 

2. Turn in assignments on the due date. 

3. Actively participate in class. 

4. Have all of my class materials with me. 

5. Do more than what is expected of me. 

Never, Once in a while, About half the time, Most of the time, Always [coded 1-5] 

.74 


a. Cronbach’s alpha (Cronbach, 1951). 

Source: Authors’ analysis of 2015 survey data from Clark County School District. 


Findings 

Academic mindsets and behaviors among 8th graders were significant predictors of 9th grade 
on-track status the following year. 

The study used logistic regression to predict the probability that students were on track 
in the 9th grade based on their 8th grade survey responses to questions about academic 
mindsets and behaviors. The results show that 8th grade students’ self-reported beliefs 
regarding growth mindset, as well as their self-reports of academic behavior, were positively 
related to the probability that they were on track at the end of the 9th grade (see appendix 
C, table C2). On the other hand, these estimates also show that student reports of perfor- 
mance avoidance, that is, the extent to which students reported hiding academic effort in 
order to avoid embarrassment or failure, were not significantly related to 9th grade on-track 
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status once measures of growth mindset and academic behavior were accounted for in 
the estimates. 

These results were used to estimate the predicted probability of being on track that is associ- 
ated with each point on the growth-mindset and academic-behavior scales (figure 6). 7 They 
show that the predicted probability of being on track at the end of the 9th grade varied 
substantially depending on students’ responses to survey questions measuring growth mind- 
set and academic behavior. The estimated probability of being on track varied from a low of 
54 percent among students with the lowest score on the 5 -point growth-mindset scale (l) to 
a high of 83 percent among students with the highest score (5) on the growth-mindset scale 
(figure 6). With respect to academic behavior, the probability of being on track for gradua- 
tion at the end of the 9th grade varied from a low of 52 percent for students with the lowest 
score on the 5-point academic-behavior scale to 86 percent for students with the highest 
score on this scale. 8 Importantly, these estimates did not control for other student charac- 
teristics and behaviors. It is not possible to know the extent to which variation in on-track 
status associated with differences in academic mindsets and behaviors was actually driven 
by differences in academic mindsets and behaviors or driven by other characteristics with 
which these measures of academic mindsets and behaviors were correlated. 


7 The predicted probability estimates are based on the model Pr(On track = l) = logit 1 

(Po + Pi Surve Yi + £i )’ 

Where Survey, is either the growth mindset or academic behavior measure taken in the spring of 
the 8th grade. 

This model is described more fully by Rabe-Hesketh and Skrondal (2012). 

8 These estimates were based on two separate logistic regressions, one of which estimated the 
relationship between 9th grade on-track status and the 8th grade measure of growth mindset, 
and one of which estimated the relationship between 9th grade on-track status and the 8th grade 
measure of academic behavior (table 4). 
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Figure 6. Predicted probability of 9th grade on-track 

status varied with 8th grade growth-mindset and 
academic-behavior scores 



Student scores on growth mindset and academic behavior scales 

Note: Predicted probability of 9th grade on-track status estimated by substituting growth mindset and 
academic behavior scores into separate logistic regressions using student scores on the 8th grade scales to 
predict 9th grade on-track status (table C2). 

Source: Authors’ analysis of primary data collected for the study. Sample size = 13,578. 
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Conclusions and implications 


The evidence presented in this report shows that 8th grade indicators can effectively iden- 
tify CCSD students who will fall off track by the end of the 9th grade. Eighth grade flags 
for attendance below 90 percent, insufficient credits, receiving Fs, and having a GPA below 
2.0 are all indicators of whether or not students will fall off track before the end of the next 
school year. With the exception of attendance, each of these indicators by itself identi- 
fied more than half of the 8th grade students who eventually fell off track at the end of the 
9th grade. Simply identifying 8th grade students who demonstrated one or more of these 
four indicators correctly detected more than three quarters (77 percent) of the students who 
fall off track in the 9th grade, before they even entered high school. If this information were 
to be disseminated to high schools prior to the beginning of the school year, high schools 
could potentially use these data to assess the extent of the challenges their incoming students 
are likely to face, as well as to devise targeted supports for high-need students that could be 
implemented in the beginning of the school year or even before they enter the 9th grade. 

The evidence in this report suggests that the indicator related to low attendance might be 
somewhat less effective than the academic indicators at identifying 8th grade students that 
are at risk of falling off track in the 9th grade. In fact, using low attendance by itself only 
identified 32 percent of the students who fell off track at the end of the 9th grade. On the 
other hand, the individual indicators for low credits, receiving at least one F, and having a 
GPA below 2.0 identified 64, 65, and 55 percent of students who fell offtrack, respectively. 
Furthermore, simple composite indicators appear to be even more effective at identifying 
a high proportion of the students who are at risk of falling off track. Thus, CCSD, and 
perhaps other districts implementing early warning systems, may wish to consider focusing 
their efforts on composite indicators. 

Each of the indicators (individual and composite) also generated false-positives, in that some 
students were identified as being at risk of falling off track but instead stayed on track. The 
results consistently show a trade-off between the percentage of off-track students correctly 
identified in the 8th grade and the false-positive rate. As the percentage of students at risk 
who are correctly identified goes up, so, too, does the percentage of students who are iden- 
tified as being at risk but who do not actually fall off track by the end of the 9th grade. In 
choosing which indicators to use, districts might consider trying to minimize false-positives 
in order to conserve resources. This may be a particularly useful approach in districts with 
limited human and financial resources. On the other hand, one cannot know in advance 
which of the identified students will or will not fall off track. Moreover, all of the variables 
used to create the indicators have been shown to be strong predictors of progress through 
high school (Balfanz, Herzog, & Maclver, 2007; Balfanz & Legters, 2004). Therefore, it is 
likely that many of the students who are identified as being at risk will, in fact, experience 
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difficulties in school and need additional support, even if they do not fall off track at the 
end of the 9th grade. For that reason, districts might want to consider erring on the side of 
casting a broader net. Practitioners involved in implementing an early indicator system will 
have to consider the costs and benefits of identifying students who may not fall off track 
versus identifying larger proportions of the students who, without any additional interven- 
tion, are likely to be off track at the end of their first year of high school. 

The evidence presented in this report also suggests that students’ self-reported beliefs 
regarding the nature of academic ability and the payoff to academic effort (growth mind- 
set), as well as their self-reports of their academic behavior at the end of the 8th grade, are 
positively related to their success in the 9th grade, specifically to their on-track status at the 
end of their first year in high school. This suggests the possibility that supports for academic 
mindsets and behaviors could be worth further investigation. This finding is consistent 
with the available evidence suggesting that academic mindsets and behaviors are potential 
drivers of the successful transition to high school (Farrington et ah, 2012; Snipes, Fancsali, 

& Stoker, 2012; Yeager, Walton, & Cohen, 2013). 

Several important cautions are worth noting. First, the evidence presented in this report 
does not identify any causal relationship between these measures of academic mindsets and 
behavior and on-track status in the following year. It is possible that other student charac- 
teristics that are not measured or accounted for in this analysis, but that are correlated with 
these measures, actually drive these estimated relationships. It is also possible that academic 
success drives these perceptions and behaviors, rather than the other way around. Additional 
research, perhaps involving short-term, low-cost randomized trials of interventions target- 
ing these beliefs and behaviors, would be necessary for sorting out any causal connections. 

Second, even to the extent that the relationships are causal, practitioners in CCSD and else- 
where should be extremely cautious regarding the use of these student survey responses as 
the basis for targeting students for support or as the basis for accountability. Tying students’ 
answers to survey questions to consequences or events (for example, being targeted for 
additional counseling support) could affect the extent to which students truthfully answer 
these questions (Duckworth, 2016). Nevertheless, the evidence presented in this report does 
suggest the possibility that strategies for supporting students’ academic mindsets and behav- 
iors may be an avenue worth investigating as a means of helping students to stay on track 
for graduation as they make the transition from the middle grades to high school. 
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Appendix A. Sample selection 
and characteristics 


To conduct the analysis described in this report, REL West obtained administrative and 
survey records collected by Clark County School District (CCSD) in the 2014/15 and 
2015/16 school years. The initial CCSD sample came from district administrative data 
and consisted of 26,322 students who were enrolled in the 8th grade in CCSD during the 
2014/15 school year and also enrolled in CCSD in the 9th grade during the 2015/16 school 
year. Students with missing data for 8th grade attendance, credits earned, or grades were 
eliminated from the analytic sample for the on-track analysis, resulting in a final analytic 
sample of 22,335 students. As can be seen in table Al, the characteristics of the students in 
this analytic sample were largely similar to those of the students in the district as a whole. 
The largest difference between the sample for the early indicators analysis and the district as 
a whole is a 4-point difference in the percentage of English learner students (14 percent for 
the student records sample versus 18 for the district average). Most of the remaining charac- 
teristics were either identical or within one percentage point of the district average. 


Table Al. Demographic characteristics of the analytic samples for 
being on track 


Demographic characteristic 
(Percentage) 

CCSD average 

Student records sample for 
early indicators analysis 

Sample for academic 
mindsets analysis 

Race/Ethnicity 

Asian or Pacific Islander 

8 

8 

9 

Black 

13 

13 

10 

Caucasian 

28 

27 

29 

Hispanic 

45 

45 

44 

Multiracial 

6 

6 

6 

Native American 

0 

1 

1 

ELL status 

English learner 

18 

14 

12 

IEP status 

IEP 

12 

10 

7 

Total N 

318,040 

22,335 

13,578 


Source: http://www.nevadareportcard.com/di/main/demoprof obtained November 4, 2016. 


In addition to the analysis of student records for the identification of early indicators of 
9th grade off-track status, this report presented an analysis of the relationship between 
9th grade off-track status and 8th grade survey measures of students’ academic mindsets 
and behaviors. In the spring of school year 2014/15, CCSD administered surveys to students 
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in grades 4-12 across the district. These surveys included a series of questions regarding 
academic mindsets and behaviors. 9 This analysis was based on a sample of students for whom 
both survey data and student records were available. Beginning with the 22,335 students 
that made up the analytic sample for the early indicators analysis, REL West selected 
those students for whom the survey data were also available. This resulted in a sample of 
13,578 students. The characteristics of the students in this sample were largely similar to 
those of the students in the district as a whole (table Al). However, the analytic sample 
for this analysis had 6 percentage points fewer English learner students than the district 
as a whole (12 percent for the academic mindsets analytic sample versus 18 for the district 
average), 5 percentage points fewer students with an individualized education plan (7 versus 
12 percent, respectively), and 3 percentage points fewer Black students (10 versus 13 percent) 
than the district as a whole. 


9 These survey measures were adapted from the Becoming Effective Learners Student (BEL-S) 
survey, originally developed by the Chicago Consortium for School Research (Farrington & 
Levenstein, 2013). 
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Appendix B. Clark County School 
District early indicator 


Though the analysis presented in this report is based on the 9th grade on-track indica- 
tor created by Allensworth and Easton (2005), it is worth pointing out that Clark County 
School District (CCSD) has developed its own definition for on-track status at the end of 
the 9th grade. According to this on-track indicator, a student is only on track at the end of 
the 9th grade if he or she successfully earns a full year credit in each of the required core 
subjects and an elective, earns half-credits for courses on computer use and health, and 
successfully passes end-of-course exams for English Language Arts I, English Language 
Arts II, Math I, and Math II. This appendix replicated the analysis of true-positives and 
false-positives using the same 8th grade indicators used in this report, but using them to 
predict whether or not students fall off track according to the CCSD 9th grade on-track 
indicator. 

The analysis of the CCSD 9th grade on-track indicator (table Bl) suggests that its use yields 
substantially lower on-track rates overall compared to the 9th grade on-track indicator 
analyzed in the main body of the report. The true-positive rates and false-positive rates 
associated with individual and composite 8th grade risk indicators were also lower than was 
the case for the on-track indicator used in the primary analysis. Some similar patterns were 
present for this indicator. The indicator based on attendance alone was associated with a 
lower true-positive rate than the indicators associated with academic performance variables. 
In addition, the true-positive rates associated with composite indicators were mostly higher 
than the true-positive rates associated with the individual indicators of risk. 
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Table Bl. 8th grade indicators and 9th grade on-track, true-positive, 
and false-positive rates for CCSD specific indicator 


8th grade indicator 

9th grade 
on-track 

True -positive 
rate 

False-positive 

rate 

Overall on-track rate 

44.9 



Attendance <90% 

22.7 

23.3 

8.4 

Credits <6 

23.6 

45.1 

17.1 

One or more Fs 

21.1 

44.1 

14.5 

GPA < 2.0 

16.0 

34.5 

8.1 

Combined indicators 

1. Attendance, Credits, or Fs 

26.6 

56.3 

25.0 

2. Attendance, Credits, or Fs (>=2 indicators) 

19.8 

42.2 

12.8 

3. Attendance, Credits, Fs, or Low GPA 

26.7 

57.8 

25.8 

4. Attendance, Credits, Fs, or Low GPA (>=2 indicators) 

20.2 

44.8 

13.9 

5. Attendance, Credits, Fs, or Low GPA (>=3 indicators) 

15.0 

32.6 

7.1 

6. Credits, Fs, or Low GPA 

24.9 

52.7 

21.4 

7. Credits, Fs, or Low GPA (>=2 indicators) 

19.4 

41.8 

12.4 


Notes: 

On-track rate = the number of 8th grade students (2014/15) who achieve on-track status in the spring of the 
9th grade (2015/16) divided by the total number of 8th grade students (2014/15) in the sample. On-track rates 
for each indicator are calculated among the students that demonstrate that indicator in the 8th grade. 

True-positive rate for each indicator is the number of students that demonstrate that indicator in the spring 
of the 8th grade (2014/15) and who are offtrack in the spring of the 9th grade (2015/16), divided by the total 
number of students that are offtrack in the spring of the 9th grade (2015/16). 

False-positive rate for each indicator is the number of students who demonstrate that indicator in the spring 
of the 8th grade (2014/15), and who are on track in the spring of the 9th grade (2015/16), divided by the total 
number of students who are on track in the spring of the 9th grade (2015/16). 
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Appendix C. Using logistic regression to 
predict on-track status 



Early indicators 

In addition to simply using dichotomous indicators of risk, it is possible to employ logistic 
regression to ascertain the extent to which student characteristics (measured as continuous 
variables) predict on- track status. To determine whether there would be any benefit to 
employing these somewhat more complex analyses compared to simply using the available 
dichotomous indicators, WestEd employed logistic regression to predict 9th grade on-track 
status on the basis of several continuous measures of 8th grade student outcomes, including 
attendance, credits, GPA, and the number of Fs. The model took the following form: 

Pr(On-track. = l) = logit -1 (|30 + ^Attendances p 2 Credits. + |3 3 Fs. + p 4 GPAl. + |3 5 GPA2. +ei), 

where, 

» On-track. = 1 if student i was on track at the end of the 9th grade (in 2015/16); 

0 otherwise; 

» Attendance. = a continuous measure of the percentage of school days (range 0 to 
100) for which student i was recorded as being in attendance during the 8th grade 
(2014/15); 

» Credits. = the number of credits earned by student i over the course of the 8th grade 
(in 2014/15) 

» Fs. = the number of semester Fs received by student i over the course of the 
8th grade school year (2015/16); 

» GPA1. = the 8th grade GPA of student i (2014/15), if the GPA was below 2.0; 

0 otherwise; and 

» GPA2. = the 8th grade GPA of student i (2014/15), if the GPA was 2.0 or above; 

0 otherwise. 10 

The parameters estimated from the data presented as odds ratios, which identify how the 
odds of achieving on-track status in grade 9 change with a one-unit change in the inde- 
pendent variable. An odds ratio of more than 1 indicates that the odds of being on track 
increase with a one-unit increase in the predictor variable. An odds ratio of less than 1 
indicates that the odds of being on track decrease with a one-unit increase in the predictor 
variable. An odds ratio of 1 indicates that the odds of achieving proficiency remain the same 
with a one-unit increase in the predictor. For example, an odds ratio estimate of 1.22 asso- 
ciated with 8th grade credits earned means that, for each 1-point increase in the number of 
9th grade credits earned, a student is 1.22 times more likely to be on track at the end of the 


10 The analysis presented in the main body of the report indicates that having a GPA below 2.0 
was associated with elevated on-track rates. Therefore, this analysis separated GPA into two 
continuous variables, one for GPAs from zero to below 2.0, and another for GPAs 2.0 and above. 
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9th grade. That is, the odds of being on track in the 9th grade increase by 22 percent with 
each one-unit increase in 8th grade credits. 


Table Cl. Logit results predicting 9th grade on-track status in 
2015/16 


Predictor (2014/15) 

Odds ratio 

Standard error 

Attendance 

1.02* 

0.003 

Credits 

1.22* 

0.047 

Fs 

1.11* 

0.028 

GPA1 

7.81* 

0.715 

GPA2 

7.16* 

0.000 


^Significant at p < .01. 

Source: Authors’ analysis of primary data collected for the study. Sample size = 26,522. 


The results (shown in table Cl) indicate that each of these 8th grade variables is a significant 
predictor of whether or not a student is off track in the 8th grade. To use this approach to 
identify whether specific students were at risk of falling off track, the study team used each 
8th grade student’s specific values for the variables included in this regression to calculate 
the predicted probability of whether or not that student was on track. WestEd then calcu- 
lated on-track rates, true-positive rates, and false-positive rates associated with different 
probability thresholds for assuming a student was offtrack (see table C2). 

The results show that, if the threshold for classifying student at risk were to be set at having 
a 50 percent or less chance of being offtrack, the on-track rate among students identi- 
fied as being at risk of falling off track would be 30 percent. This approach would have a 
true-positive identify 54 percent (that is, it would successfully identify 54 of the 8th graders 
who would eventually fall off track in the 9th grade). It would also generate a 9.3 percent 
false-positive rate. The rest of the table shows how these numbers change when different 
threshold probabilities for identifying students as at risk of being offtrack are selected. 

For example, if the threshold for identification is moved to include any students who have 
greater than a 40 percent chance of falling off track, the on-track rate for identified students 
becomes 36 percent, the percentage of off-track students correctly flagged by this indicator 
becomes 68 percent, and the false-positive rate becomes 15 percent. When the threshold 
for identification is reduced to students with a greater than 30 percent predicted probabil- 
ity of falling off track, the actual 9th grade on-track rate among identified students is 42 
percent, the percentage of off-track students correctly identified becomes 77 percent, and 
the false-positive rate is 22 percent. 
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Table C2. On-track, true-positive, and false-positive rates based on 
logistic regression 


Probability threshold 

9th grade on-track rate 

True -positive rate 

False -positive rate 

>80% probability off track) 

15.8 

19.9 

1.5 

>70% probability off track) 

19.0 

31.9 

3.1 

> 60% probability off track) 

24.7 

42.9 

5.7 

> 50% probability off track) 

29.6 

54.3 

9.3 

> 40% probability off track) 

35.8 

67.7 

15.4 

> 30% probability off track) 

41.5 

77.4 

22.4 

> 20% probability off track) 

48.2 

86.7 

33.0 


Source: Authors’ analysis of primary data collected for the study. Sample size = 26,522. 


This is somewhat more efficient than the simpler approach based on the dichotomous flags. 
When using that approach (table l), the option with the highest correct-identification rate 
was the indicator based on flags for 8th grade attendance, credits, Fs, and GPA. Identifying 
8th grade students based on this flag resulted in on-track rates of 49 percent among identi- 
fied students, correctly identified 77 percent of students who fell off track by the end of the 
9th grade, and resulted in a false-positive rate of 30 percent, 8 percentage points higher than 
the false-positive rate based on the logit approach. In short, the approach based on the logit 
appears to produce slightly more efficient results and is less likely to falsely identify students 
who, in reality, are not going to fall off track. On the other hand, the approach based on the 
dichotomous flags is easier to implement. States and districts implementing on-track systems 
will have to decide if the reductions in the false-positive rate are, in fact, desirable and worth 
the increased complexities associated with the approach based on the logistic regression. 

Academic mindsets logistic regression 

The study also employed logistic regression to examine the extent to which students’ 
responses to 8th grade survey questions about academic mindsets and behaviors predicted 
whether or not students were on track in the 9th grade. The model took the following basic 
form: 

Pr(On-track. = l) = logit _1 (p0 + ^growth mindset.+ p 2 academic behaviors. + ^performance 
avoidance. + e.), 

where growth mindset., academic behaviors., and performance avoidance., are survey 
measures for student i collected in the spring of the 8th grade (2014/15), and are defined as 
described in table 3. Different versions of the model, with different survey measures included 
and excluded, were estimated for this analysis (see table C2). 

The results show that 8th grade students’ reported beliefs regarding growth mindset, as well 
as their self-reports of academic behavior, were positively related to the probability that they 
were on track at the end of the 9th grade (table C3). On the other hand, these estimates also 
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show that student reports of performance avoidance, that is, the extent to which students 
reported hiding academic effort in order to avoid embarrassment or failure, were not signifi- 
cantly related to 9th grade on-track status, once measures of growth mindset and academic 
behavior were accounted for in the estimates. Though these estimates do not provide a basis 
for assessing whether or not these attitudes and behaviors are causally related to 9th grade 
on-track status, the presence of significant relationships between these survey measures 
both validates the measures themselves and suggests that additional research is warranted to 
examine the extent to which these beliefs and behaviors can be targeted and changed and 
whether such changes result in improved transitions to high school. 


Table C3. Logistic regression: 9th grade on-track status and 

8th grade academic mindset and behavior survey measures 


Statistic 

Growth mindset 

Academic 

behavior 

Growth mindset 
and academic 
behavior 

Growth mindset, 
academic 
behavior, and 
performance 

Odds ratio (standard errors) 

Growth mindset 

1.52* (.003) 


1.41* (0.03) 

1.44* (0.03) 

Academic behavior 


1.98* (0.06) 

1.82* (0.05) 

1.83* (0.05) 

Performance avoidance 




1.04 (0.03) 


*Significant at p < 0.01. 

N= 13,578. 

Source: Authors’ analysis of primary data collected for the study. Sample size = 13,578. 
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